A Unified Approach to Mathematical Optimization and Lagrange Multiplier Theory for Scientists and Engineers

It should be no surprise that the differentiation of functionals (real valued functions) defined on abstract spaces plays a fundamental role in continuous optimization theory and applications. Differential tools were the major part of the early calculus of variations and actually it was their use that motivated the terminology calculus of variations. The tools that we develop that allow us to work in the full generality of vector spaces are the so-called one-sided directional variation, the directional variation, and the directional derivative. Their development will be our first task. In some theory and applications we need more powerful differentiation tools. Towards this end our second task will be the presentation of the Gâteaux and Fréchet derivatives. Since these derivatives offer more powerful tools, their definition and use will require stronger structure than just vector space structure. The price that we have to pay is that we will require a normed linear space structure. While normed linear spaces are not as general as vector spaces, the notion of the Fréchet derivative in a normed linear space is a useful theoretical tool and carries with it a very satisfying theory. Hence, it is the preferred differentiation notion in theoretical mathematics. Moreover, at times in our optimization applications we will have to turn to this elegant theory. However, we stress that the less elegant differentiation notions will often lead us to surprisingly general and useful applications; therefore, we promote and embrace the study of all these theories. We stress that the various differentiation notions that we present are not made one bit simpler, or proofs shorter, if we restrict our attention to finite dimensional spaces, indeed to IR. Dimension of spaces is not an inherent part of differentiation when properly presented. However, we quickly add that we will often turn to IR for examples because of its familiarity and rich base of examples. The reader interested in consulting references on differentiation will find a host of treatments in elementary analysis books in the literature. However, we have been guided mostly by Ortega and Rheinboldt [2]. Our treatment is not the same as theirs but it is similar. In consideration of the many students that will read our presentation, we have included numerous examples throughout our presentation and have given them in unusually complete detail.